Our client, a leading multinational pharmaceutical company based in Copenhagen, Denmark, sought to revamp their digital transformation journey and improve customer, partner and employee experiences across their brands worldwide. HCLTech partnered with them to design a unified platform aimed at transforming their R&D technology area. Leveraging Databricks workflows, HCLTech built data pipelines to deliver data migration services and a framework to enable digital transformation through unified platform.
The Challenge
Overcoming legacy systems and data silos
Our client faced numerous challenges, including multiple data silos, processing inefficiencies, limited digitization and data tracking capabilities in their legacy systems. Additionally, they struggled with data sharing processes, compliance with GDPR, data quality and storage, real-time access and high-risk management.
The Objective
Enhance data processing and compliance with a reusable framework
Our client aimed to establish a reusable data migration framework for clinical trial data and documents, transitioning from on-premises systems to Veeva Vault, a unified platform program. The strategic initiative was designed to accelerate data processing for downstream machine learning, bolster data security to comply with Health Insurance Portability and Accountability Act (HIPAA) requirements and perform data transformation in the R&D technology area. Our client sought a solution that could provide higher visibility, collaboration and flexibility, reducing development and testing time while ensuring built-in error and audit capture mechanisms.
The Solution
Optimizing performance with a custom-built architecture
Our team tackled these challenges by harnessing the capabilities of Azure Databricks. We employed this technology to orchestrate dependable data pipelines and develop a sophisticated, four-layered modular architecture tailored to our client's needs:
- Data ingestion: The first layer focused on efficiently gathering data from various sources, ensuring a smooth and seamless process of data collection
- Data cleansing: The second layer involved thorough data cleansing procedures, eliminating inconsistencies and errors to enhance the overall quality and reliability of the data
- Data transformation: The third layer was dedicated to transforming the data into usable formats, optimizing it for analysis and decision-making purposes
- Dispatch: The fourth layer handled the distribution of processed data to its intended destinations, ensuring timely delivery and accessibility
This robust and adaptable framework offered several key benefits:
- Operational dashboards: Providing comprehensive dashboards for monitoring and managing data pipelines in real-time, enhancing operational efficiency and decision-making
- Real-time visibility: Offering insights into the status and performance of data pipelines, enabling proactive identification and resolution of issues
- GDPR compliance: Ensuring adherence to GDPR regulations by implementing robust data governance and security measures throughout the data lifecycle
- Scalability: Providing a scalable solution capable of accommodating evolving data migration needs and scaling seamlessly to handle increased volumes of data
Overall, our approach provided our client with a reliable, efficient and GDPR-compliant solution that not only addressed immediate challenges but also laid the foundation for future data migration endeavors.
The Impact
Accelerated development and improved scalability
Our solution involving Databricks resulted in a 40% reduction in development and unit testing efforts, faster time to market and lower TCO using HCLTech's proprietary framework. With prebuilt data pipeline patterns covering around 80% of data pipeline requirements, the adaptive, repeatable and easily maneuverable factory-driven model is future-ready, cloud-neutral and capable of supporting IoT data and various data formats.
